Entropy Maximization for Partially Observable Markov Decision Processes

We study the problem of synthesizing a controller that maximizes entropy partially observable Markov decision process (POMDP) subject to constraint on expected total reward. Such minimizes predictability an agent’s trajectories outside observer while guaranteeing completion task expressed by reward function. Focusing finite-state controllers (FSCs) with deterministic memory transitions, we show maximum POMDP is lower bounded parameteric chain (pMC) induced such FSCs. This relationship allows us recast maximization as so-called parameter synthesis for pMC. then present algorithm synthesize FSC locally over FSCs same number states. In numerical example, highlight benefit using entropy-maximizing compared simply finds feasible policy accomplishing task.